Informal versus formal judgment of statistical models: The case of normality assumptions

Researchers sometimes use informal judgment for statistical model diagnostics and assumption checking. Informal might seem more desirable than formal because of a paradox: Formal hypothesis tests assumptions appear to become less useful as sample size increases. We suggest that this paradox can be resolved by evaluating both via simplified signal detection framework. In 4 studies, we used approach compare judgments normality diagnostic graphs (histograms, Q–Q plots, P–P plots) the performance several (Shapiro–Wilk test, Kolmogorov–Smirnov etc.). Participants judged whether or not data came from normal population (Experiments 1–2) close enough parametric test powerful nonparametric one 3–4). Across all experiments, participants’ showed lower discriminability did tests. This pattern occurred even after participants were given 400 training trials with feedback, financial incentive, ecologically valid distribution shapes. The advantage led slightly follow-up (parametric vs. nonparametric). Overall, framework here suggests may ones.